1. Field of the Invention
The present invention relates to cameras. More specifically, the present invention relates the design of a “plenoptic” camera, which captures information about the direction distribution of light rays entering the camera.
2. Related Art
Conventional cameras fail to capture a large amount of optical information. In particular, a conventional camera does not capture information about the location of the aperture of the light rays entering the camera. During operation, a conventional digital camera captures a two-dimensional (2D) image representing a total amount of light which strikes each point on a photosensor within the camera. However, this 2D image contains no information about the directional distribution of the light that strikes the photosensor. This directional information at the pixels corresponds to locational information at the aperture.
In contrast, a “plenoptic” camera samples the four-dimensional (4D) optical phase space or light field and in doing so captures information about the directional distribution of the light rays. For example, see [Adelson92] Adelson, T., and Wang, J. Y. A. 1992, “Single lens stereo with a plenoptic camera,” IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 2, February 1992, pp. 99-106. Also see [Ng05] Ng, R., Levoy, M., Bredif, M., Duval, G., Horowitz, M. and Hanrahan, P., “Light Field Photography with a Hand-Held Plenoptic Camera,” Stanford University Computer Science Tech Report CSTR 2005-02, April 2005. These papers describe plenoptic/light-field camera designs based on modifications to a conventional digital camera.
Referring to FIG. 1A, the system described in [Ng05] uses a microlens array 106 comprised of about 100,000 lenslets which is placed a small distance (0.5 mm) from a CCD array 108. Each lenslet splits a beam coming to it from the main lens 104 into (100) rays coming from different “pinhole” locations on the aperture the main lens 104. Each of these rays is recorded as a pixel, and the pixels under each lenslet collectively form a 100-pixel image. If we call this 100-pixel image a “macropixel,” then the plenoptic photograph captured by this camera will contain approximately 100,000 macropixels. By appropriately selecting a pixel from each macropixel, we can create conventional pictures taken with a virtual pinhole camera. Moreover, by mixing such images appropriately, we can refocus images originally taken out-of-focus, reduce noise, or achieve other “light-field” effects, as described in the papers above.
In the prototype described in [Ng05], a 16-megapixel sensor is used with an approximately 100,000 lenslet array to create a final output of approximately 300×300 macropixels, with one macropixel per lenslet. The macropixel created by each lenslet comprises approximately 150 pixels. However, only about 100 of these pixels are useful because of poor quality of edge pixels caused by a problem which is referred to as “vignetting.” These 100 pixels which comprise each macropixel make the captured data equivalent to 100 conventional images, one for each choice of the pixel inside a macropixel. The size of each picture produced by processing data from this camera is equal to the number of lenslets, and is hence 300×300.
Unfortunately, an image with only 300×300 pixels has insufficient resolution for most practical uses. The number of pixels can be increased by increasing the number of lenslets and making them smaller. Unfortunately, the prior art cannot use the border pixels of each image. Note that a band of about 2 to 4 pixels along the border of the macropixel is lost depending upon whether the system is working with a Grayscale pattern or a Bayer pattern. When the image is small, these few border pixels comprise a large percentage of the image. For example, in a 10×10 color image, 4 pixels on each edge may be lost leaving only 2×2=4 central pixels. In this case, 96% of the information lost! Because of this problem, the system described in [Ng05] cannot reduce the size of each microlens and the image under it. Consequently, the number of microlenses, and hence the resolution of the image, is limited. (Currently, in a system that uses a 16-megapixel sensor, the number of microlenses is limited to less than 100,000.)
Hence, what is needed is a method and an apparatus for increasing the resolution of a plenoptic camera without the above-described problems.